{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用字符级RNN生成姓名"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们仍然使用手工搭建的包含几个线性层的小型RNN。与之前的预测姓名最大的区别是：它不是“阅读”输入的所有字符然后生成一个预测分类，而是输入一个分类然后在每个时间步生成一个字母。循环预测字母来形成一个语言的语句通常被视作**语言模型**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">数据下载通道: 点击[这里](https://download.pytorch.org/tutorial/data.zip)下载数据集。解压到当前工作目录。\n",
    "\n",
    "就和上个预测姓名分类的教程一样，我们有一个姓名文件夹 `data/names/[language].txt` ，每个姓名一行。我们将它转化为一个 array, 转为ASCII字符，最后生成一个字典 `{language: [name1, name2,...]}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# categories: 18 ['Arabic', 'Chinese', 'Czech', 'Dutch', 'English', 'French', 'German', 'Greek', 'Irish', 'Italian', 'Japanese', 'Korean', 'Polish', 'Portuguese', 'Russian', 'Scottish', 'Spanish', 'Vietnamese']\n",
      "O'Neal\n"
     ]
    }
   ],
   "source": [
    "from __future__ import unicode_literals, print_function, division\n",
    "from io import open\n",
    "import glob\n",
    "import os\n",
    "import unicodedata\n",
    "import string\n",
    "\n",
    "all_letters = string.ascii_letters + \" .,;'-\"\n",
    "n_letters = len(all_letters) + 1 # Plus EOS marker\n",
    "\n",
    "def findFiles(path): return glob.glob(path)\n",
    "\n",
    "# Turn a Unicode string to plain ASCII, thanks to http://stackoverflow.com/a/518232/2809427\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "        and c in all_letters\n",
    "    )\n",
    "\n",
    "# Read a file and split into lines\n",
    "def readLines(filename):\n",
    "    lines = open(filename, encoding='utf-8').read().strip().split('\\n')\n",
    "    return [unicodeToAscii(line) for line in lines]\n",
    "\n",
    "# Build the category_lines dictionary, a list of lines per category\n",
    "category_lines = {}\n",
    "all_categories = []\n",
    "for filename in findFiles('data/names/*.txt'):\n",
    "    category = os.path.splitext(os.path.basename(filename))[0]\n",
    "    all_categories.append(category)\n",
    "    lines = readLines(filename)\n",
    "    category_lines[category] = lines\n",
    "\n",
    "n_categories = len(all_categories)\n",
    "\n",
    "if n_categories == 0:\n",
    "    raise RuntimeError('Data not found. Make sure that you downloaded data '\n",
    "        'from https://download.pytorch.org/tutorial/data.zip and extract it to '\n",
    "        'the current directory.')\n",
    "\n",
    "print('# categories:', n_categories, all_categories)\n",
    "print(unicodeToAscii(\"O'Néàl\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. 搭建网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "新的网络结果扩充了姓名识别的RNN网络，它的输入增加了一个分类Tensor，该张量同样参与与其他输入的结合(concatenate)。分类张量也是一个one-hot向量。\n",
    "\n",
    "我们将输出解释为下一个字母的概率。采样时，最可能的输出字母用作下一个输入字母。 \n",
    "\n",
    "同时，模型增加了第二个线性层(在隐藏层的输出组合之后)，从而增强其性能。后续一个 dropout 层，它随机将输入置0(这里的概率设置为0.1),一般用来模糊输入来达到规避过拟合的问题。在这里，我们将它用于网络的末端，故意添加一些混乱进而增加采样种类。  \n",
    "\n",
    "网络模型如下所示：\n",
    "![](https://i.imgur.com/jzVrf7f.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN,self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        \n",
    "        self.i2h = nn.Linear(n_categories + input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(n_categories + input_size + hidden_size, output_size)\n",
    "        self.o2o = nn.Linear(hidden_size + output_size, output_size)\n",
    "        self.dropout = nn.Dropout(0.1)\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "        \n",
    "    def forward(self, category, input, hidden):\n",
    "        input_combined = torch.cat([category, input, hidden],dim=1)\n",
    "        hidden = self.i2h(input_combined)\n",
    "        output = self.i2o(input_combined)\n",
    "        output_combined = torch.cat([hidden,output],1)\n",
    "        output = self.o2o(output_combined)\n",
    "        output = self.dropout(output)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "    \n",
    "    def initHidden(self):\n",
    "        return torch.zeros(1, self.hidden_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 训练准备\n",
    "首先，辅助函数用来获取(category, line)对"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# Random item from a list\n",
    "def randomChoice(l):\n",
    "    return l[random.randint(0, len(l)-1)]\n",
    "\n",
    "# Get a random category and random line from that category\n",
    "def randomTrainingPair():\n",
    "    category = randomChoice(all_categories)\n",
    "    line = randomChoice(category_lines[category])\n",
    "    return category, line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于每个时间步(训练词语的每个字母)，网络的输入为 `(category, current letter, hidden state)`， 输出为 `(next letter, next hidden state)`。因此对于每个训练集，我们需要一个分类，一个输入字母集合，还有一个目标字母集合。\n",
    "\n",
    "由于我们需要在每个时间步通过当前字母来预测下一个字母，字母对的形式应该类似于这样，比如 `\"ABCD<EOS>\"` , 则我们会构建('A','B'),('B','C'),('C','D'),('D','E'),('E','EOS')。  \n",
    "\n",
    "用图来表示如下：\n",
    "![](https://i.imgur.com/JH58tXY.png)\n",
    "\n",
    "分类张量是一个one-hot张量，大小为 `<1 x n_categories>`。在训练的每个时间步我们都将其作为输入。这是众多设计选择的一个，它同样可以作为初始隐藏状态或其他策略的一部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# one-hot vector for category\n",
    "def categoryTensor(category):\n",
    "    li = all_categories.index(category)\n",
    "    tensor = torch.zeros(1, n_categories)\n",
    "    tensor[0][li]=1\n",
    "    return tensor\n",
    "\n",
    "# one-hot matrix of first to last letters (not including EOS) for input\n",
    "def inputTensor(line):\n",
    "    tensor = torch.zeros(len(line),1, n_letters)\n",
    "    for li in range(len(line)):\n",
    "        letter = line[li]\n",
    "        tensor[li][0][all_letters.find(letter)]=1\n",
    "    return tensor\n",
    "\n",
    "# LongTensor of second letter to end(EOS) for target\n",
    "def targetTensor(line):\n",
    "    letter_indexes = [all_letters.find(line[li]) for li in range(1, len(line))]\n",
    "    letter_indexes.append(n_letters-1)  # EOS\n",
    "    return torch.LongTensor(letter_indexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "方便起见，在训练过程中我们使用`randomTrainingExample` 函数来获取一个随机的 (category, line) 对，然后将其转化为输入要求的 (category, input, target) 张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# make category, input, and target tensors from a random category, line pair\n",
    "def randomTrainingExample():\n",
    "    category, line = randomTrainingPair()\n",
    "    category_tensor = categoryTensor(category)\n",
    "    input_line_tensor = inputTensor(line)\n",
    "    target_line_tensor = targetTensor(line)\n",
    "    return category_tensor, input_line_tensor, target_line_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 训练网络\n",
    "与分类相反，分类仅仅使用最后一层输出，这里我们使用每个时间步的输出作为预测，所以我们需要计算每个时间步的损失\n",
    "\n",
    "autograd 的魔力使你能够简单的将所有时间步的loss相加，然后在最后反向传播"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "criterion = nn.NLLLoss()\n",
    "\n",
    "learning_rate = 0.0005\n",
    "\n",
    "def train(category_tensor, input_line_tensor, target_line_tensor):\n",
    "    target_line_tensor.unsqueeze_(-1)\n",
    "    hidden = rnn.initHidden()\n",
    "    \n",
    "    rnn.zero_grad()\n",
    "    \n",
    "    loss = 0\n",
    "    \n",
    "    for i in range(input_line_tensor.size(0)):\n",
    "        output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)\n",
    "        l = criterion(output, target_line_tensor[i])\n",
    "        loss+=l\n",
    "        \n",
    "    loss.backward()\n",
    "    \n",
    "    for p in rnn.parameters():\n",
    "        p.data.add_(-learning_rate, p.grad.data)\n",
    "        \n",
    "    return output, loss.item() / input_line_tensor.size(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了跟踪训练时间，这里添加了一个 `timeSince(timestep)`函数，该函数返回一个可读字符串"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "def timeSince(since):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    m = math.floor(s/60)\n",
    "    s -= m*60\n",
    "    return '%dm %ds' %(m,s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练依旧很花时间-调用训练函数多次，并在每个 `print_every` 样本后打印损失，同时在每个 `plot_every` 样本后保存损失到 `all_losses` 方便后续的可视化损失"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0m 17s (5000 5%) 2.1339\n",
      "0m 34s (10000 10%) 2.3110\n",
      "0m 53s (15000 15%) 2.2874\n",
      "1m 13s (20000 20%) 3.5956\n",
      "1m 33s (25000 25%) 2.4674\n",
      "1m 52s (30000 30%) 2.3219\n",
      "2m 9s (35000 35%) 3.0257\n",
      "2m 27s (40000 40%) 2.5090\n",
      "2m 45s (45000 45%) 1.9921\n",
      "3m 4s (50000 50%) 2.0124\n",
      "3m 22s (55000 55%) 2.8580\n",
      "3m 41s (60000 60%) 2.4451\n",
      "3m 59s (65000 65%) 3.1174\n",
      "4m 16s (70000 70%) 1.7301\n",
      "4m 34s (75000 75%) 2.9455\n",
      "4m 52s (80000 80%) 2.3166\n",
      "5m 9s (85000 85%) 1.2998\n",
      "5m 27s (90000 90%) 2.1184\n",
      "5m 45s (95000 95%) 2.6679\n",
      "6m 3s (100000 100%) 2.4100\n"
     ]
    }
   ],
   "source": [
    "rnn = RNN(n_letters, 128, n_letters)\n",
    "\n",
    "n_iters = 100000\n",
    "print_every = 5000\n",
    "plot_every = 500\n",
    "all_losses = []\n",
    "total_loss = 0 # Reset every plot_every iters\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "for iter in range(1, n_iters + 1):\n",
    "    output, loss = train(*randomTrainingExample())\n",
    "    total_loss += loss\n",
    "\n",
    "    if iter % print_every == 0:\n",
    "        print('%s (%d %d%%) %.4f' % (timeSince(start), iter, iter / n_iters * 100, loss))\n",
    "\n",
    "    if iter % plot_every == 0:\n",
    "        all_losses.append(total_loss / plot_every)\n",
    "        total_loss = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 打印损失"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1acb9d51b70>]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAFkCAYAAACjCwibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYVOXZx/HvvSBNBSMqWEAlgGJ5MaxiBUEFsUdRw4JR\niSSvKG8MamIPaqzR2AJijWBbC8Yo9oKKCoouikbBBgoWIFiWIouU+/3j3snMLltmtjCzy+9zXXPt\nzjnPmfPMHtj57XOeYu6OiIiISHXysl0BERERaRgUGkRERCQtCg0iIiKSFoUGERERSYtCg4iIiKRF\noUFERETSotAgIiIiaVFoEBERkbQoNIiIiEhaFBpEREQkLbUKDWZ2rpmtMbPrqinXx8yKzKzEzD42\ns5Nqc14RERFZ92ocGsxsD+B3wIxqym0HPAG8CHQHbgTuMLN+NT23iIiIrHs1Cg1mthFwLzAM+KGa\n4sOB2e7+J3f/yN3HABOAkTU5t4iIiGRHTVsaxgAT3X1SGmX3Al4ot+1ZYO8anltERESyoGmmB5jZ\nIGA3YPc0D2kPLCi3bQHQ2syau/uKCs7RFjgY+BwoybSOIiIi67EWwHbAs+7+bV2+cEahwcy2AW4A\nDnL3lXVZkXIOBu6rx9cXERFp7IYA99flC2ba0pAPbA5MNzMr3dYE6G1mI4Dm7u7ljpkPtCu3rR2w\nuKJWhlKfA9x7771069YtwypKLho5ciTXX399tqshdUTXs3HR9WxcZs6cyQknnACln6V1KdPQ8AKw\na7lt44CZwFUVBAaAqcAh5bb1L91emRKAbt260aNHjwyrKLmoTZs2upaNiK5n46Lr2WjV+e39jEKD\nuy8DPkzdZmbLgG/dfWbp8yuArd09MRfDLcDpZnY18A/gQOBY4NBa1l1ERETWobqYEbJ868KWQIf/\n7nT/HDgMOAh4lxhqeYq7lx9RISIiIjks49ET5bn7AeWeD62gzGSiP4SIiIg0UFp7QtaJgoKCbFdB\n6pCuZ+Oi6ynpUmiQdUK/lBoXXc/GRddT0qXQICIiImlRaBAREZG0KDSIiIhIWhQaREREJC0KDSIi\nIpIWhQYRERFJi0KDiIiIpEWhQURERNKi0CAiIiJpUWgQERGRtCg0iIiISFoUGkRERCQtCg0iIiKS\nFoUGERERSUtOh4bVq7NdAxEREUnI6dCwYkW2ayAiIiIJCg0iIiKSlpwODSUl2a6BiIiIJCg0iIiI\nSFoUGkRERCQtCg0iIiKSloxCg5mdamYzzKy49DHFzAZUc8wQM3vXzJaZ2ddmdqeZbZrO+dQRUkRE\nJHdk2tIwDzgH6AHkA5OAx8ysW0WFzWxfYDxwO7ATcCzQE7gtnZOppUFERCR3NM2ksLs/WW7ThWY2\nHNgLmFnBIXsBc9x9TOnzL8zsVuBP6ZxPLQ0iIiK5o8Z9Gswsz8wGAa2AqZUUmwp0MLNDSo9pBxwH\nlA8fFVJLg4iISO7IqKUBwMx2IcJAC2AJcLS7z6qorLtPMbMTgAfNrEXp+R4HRqRzLrU0iIiI5I6a\ntDTMAroTfRPGAneb2Y4VFTSznYAbgYuJfhAHA9sDt6ZzouXLa1A7ERERqRfm7rV7AbPngU/dfXgF\n++4GWrj78Snb9gVeBbZ09wWVvGYPoKhjx950796mzL6CggIKCgpqVWcREZHGoLCwkMLCwjLbiouL\nmTx5MkC+u0+vy/NlfHuiAnlA80r2tQJ+KrdtDeCAVffCfftez7hxPWpXOxERkUaqoj+kp0+fTn5+\nfr2cL6PQYGZXAE8Dc4GNgSHA/kD/0v1XAlu5+0mlh0wEbjOzU4Fnga2A64E33X1+dedTR0gREZHc\nkWlLwxbEvAtbAsXAe0B/d59Uur890CFR2N3Hm9lGwOnAtcAPwIvAuemcTB0hRUREckem8zQMq2b/\n0Aq2jQHGVFC8WmppEBERyR1ae0JERETSktOhQbcnREREckdOhwa1NIiIiOQOhQYRERFJi0KDiIiI\npCWnQ4P6NIiIiOSOnA4NamkQERHJHQoNIiIikpacDg26PSEiIpI7cjo0rF4NK1dmuxYiIiICOR4a\nAJYvz3YNREREBBpAaPjxx2zXQEREREChQURERNKU86FBtydERERyQ86HBrU0iIiI5IacDw1qaRAR\nEckNOR8a1NIgIiKSG3I+NKilQUREJDfkfGhQS4OIiEhuUGgQERGRtOR0aGjeXLcnREREckVOh4YW\nLdTSICIikityOjSopUFERCR35HRoUEuDiIhI7sgoNJjZqWY2w8yKSx9TzGxANcc0M7PLzexzMysx\ns9lmdnI651NoEBERyR1NMyw/DzgH+AQw4GTgMTPbzd1nVnLMw8DmwFDgM2BL0gwruj0hIiKSOzIK\nDe7+ZLlNF5rZcGAvYK3QUNoK0Qvo5O4/lG6em+751NIgIiKSO2rcp8HM8sxsENAKmFpJsSOAt4Fz\nzOxLM/vIzK4xsxbpnEOhQUREJHdkensCM9uFCAktgCXA0e4+q5LinYiWhhLgl8BmwFhgU+CU6s7V\nooVuT4iIiOSKjEMDMAvoDrQBjgXuNrPelQSHPGANMNjdlwKY2ZnAw2Z2mruvqOpE7703kp9+asOR\nRya3FRQUUFBQUINqi4iINC6FhYUUFhaW2VZcXFxv5zN3r90LmD0PfOruwyvYNw7Yx927pmzbEfgA\n6Orun1Xymj2AouOOK+KTT3rwzju1qqKIiMh6Y/r06eTn5wPku/v0unztupinIQ9oXsm+14GtzKxV\nyrYdiNaHL6t7Yd2eEBERyR2ZztNwhZn1MrNtzWwXM7sS2B+4t3T/lWY2PuWQ+4FvgbvMrJuZ9Qb+\nCtxZ3a0JUEdIERGRXJJpn4YtgPHEXAvFwHtAf3efVLq/PdAhUdjdl5lZP+DvwFtEgHgQuCidkzVv\nrtAgIiKSKzKdp2FYNfuHVrDtY+DgDOsFqKVBREQkl+T82hPLl8OaNdmuiYiIiOR8aAAoKcluPURE\nRKSBhAaNoBAREcm+nA4NzUsHcqpfg4iISPbldGhItDQoNIiIiGRfToeGli3jq0KDiIhI9uV0aFBL\ng4iISO5QaBAREZG0NIjQoNETIiIi2ZfToUGjJ0RERHKHQoOIiIikJadDQ5MmWrRKREQkV+R0aABo\n1UqhQUREJBcoNIiIiEhaGkRo0OgJERGR7Mv50NCypVoaREREckHOhwbdnhAREckNCg0iIiKSFoUG\nERERSUuDCA3qCCkiIpJ9OR8a1BFSREQkN+R8aNDtCRERkdyg0CAiIiJpySg0mNmpZjbDzIpLH1PM\nbECax+5rZivNbHom51RoEBERyQ2ZtjTMA84BegD5wCTgMTPrVtVBZtYGGA+8kGkFFRpERERyQ0ah\nwd2fdPdn3P0zd//U3S8ElgJ7VXPoLcB9wBuZVlCjJ0RERHJDjfs0mFmemQ0CWgFTqyg3FNgeuKQm\n50mMnnCvWT1FRESkbjTN9AAz24UICS2AJcDR7j6rkrJdgCuA/dx9jZllXMFWreJrSUkECBEREcmO\nmrQ0zAK6Az2BscDdZrZj+UJmlkfckhjl7p8lNmd6skRoUL8GERGR7DKvZbu/mT0PfOruw8ttbwN8\nD6wiGRbySr9fBfR395crec0eQFHv3r1ZsaINb74JBx0ULQ0FBQUUFBTUqs4iIiKNQWFhIYWFhWW2\nFRcXM3nyZIB8d89oxGJ16iI0vAh84e6/KbfdgPKjKk4H+gIDgc/dvcIujonQUFRUxLJlPejdG2bN\ngh12qFVVRUREGr3p06eTn58P9RAaMurTYGZXAE8Dc4GNgSHA/kD/0v1XAlu5+0keaeTDcscvBErc\nfWa650zcntAIChERkezKtCPkFsR8C1sCxcB7xG2GSaX72wMd6q56yc6P6tMgIiKSXRmFBncfVs3+\nodXsv4QMh16qI6SIiEhuaBBrT4BCg4iISLYpNIiIiEhacj40qE+DiIhIbsj50NCkCTRrptETIiIi\n2ZbzoQG00qWIiEguUGgQERGRtCg0iIiISFoUGkRERCQtCg0iIiKSlgYRGlq21OgJERGRbGsQoUEt\nDSIiItmn0CAiIiJpUWgQERGRtCg0iIiISFoUGkRERCQtDSI0aPSEiIhI9jWI0KCWBhERkexTaBAR\nEZG0NKjQ4J7tmoiIiKy/GkRoaNMmvv7wQ3brISIisj5rEKFh++3j65w52a2HiIjI+qxBhYbZs7Nb\nDxERkfVZgwgNbdvCxhurpUFERCSbGkRoMINOndTSICIikk0ZhQYzO9XMZphZceljipkNqKL80Wb2\nnJktTCnfvyYVVWgQERHJrkxbGuYB5wA9gHxgEvCYmXWrpHxv4DngkNJjXgImmln3TCuq0CAiIpJd\nTTMp7O5Pltt0oZkNB/YCZlZQfmS5TReY2VHAEcCMTM69/fbwxRewejU0aZLJkSIiIlIXatynwczy\nzGwQ0AqYmuYxBmwMfJfp+Tp1gpUr4auvMj1SRERE6kLGocHMdjGzJcAK4GbgaHeflebhfwQ2BB7K\n9LydOsVX3aIQERHJjoxuT5SaBXQH2gDHAnebWe/qgoOZDQYuAo5090XpnGjkyJG0KZ0Ocs2a2FZY\nWECfPgU1qLaIiEjjUlhYSGFhYZltxcXF9XY+81ou6GBmzwOfuvvwKsoMAu4AjnX3Z9J4zR5AUVFR\nET169Pjv9m22gaFD4S9/qVWVRUREGq3p06eTn58PkO/u0+vytetinoY8oHllO82sALgTGJROYKiK\nRlCIiIhkT0a3J8zsCuBpYC7RoXEIsD/Qv3T/lcBW7n5S6fPBwDjg98BbZtau9KWWu/viTCvbqRN8\n9FGmR4mIiEhdyLSlYQtgPNGv4QVirob+7j6pdH97oENK+d8CTYAxwNcpjxtqUtlOnTSVtIiISLZk\nOk/DsGr2Dy33vG9NKlWZ7beHBQtg2TLYcMO6fGURERGpToNYeyIhMexSrQ0iIiLrXoMMDeoMKSIi\nsu41qNDQvj20agWffprtmoiIiKx/GlRoMIPOneGTT7JdExERkfVPgwoNAF27wscfZ7sWIiIi658G\nFxq6dFFLg4iISDY0yNAwbx4sX57tmoiIiKxfGmRoAPjss+zWQ0REZH3T4EJD167xVf0aRERE1q0G\nFxo23xxat1a/BhERkXWtwYUGM3WGFBERyYYGFxpAoUFERCQbGmRo0FwNIiIi616DDA1dusD8+bBk\nSbZrIiIisv5osKEBtAaFiIjIutSgQ4P6NYiIiKw7DTI0bLoptG2rfg0iIiLrUoMMDRCtDe++m+1a\niIiIrD8abGg44QR45BF4/vls10RERGT90GBDw/DhcNBBcPLJ8N132a6NiIhI49dgQ0NeHtx1F/z4\nI5x+erZrIyIi0vg12NAAsM028Pe/wwMPwGuvZbs2IiIijVuDDg0AgwfDbrvBBReAe7ZrIyIi0nhl\nFBrM7FQzm2FmxaWPKWY2oJpj+phZkZmVmNnHZnZS7apcVl4eXHYZTJ6sTpEiIiL1KdOWhnnAOUAP\nIB+YBDxmZt0qKmxm2wFPAC8C3YEbgTvMrF8N61uhQw+FvfdWa4OIiEh9yig0uPuT7v6Mu3/m7p+6\n+4XAUmCvSg4ZDsx29z+5+0fuPgaYAIysXbXLMoPLL4e334YnnqjLVxYREZGEGvdpMLM8MxsEtAKm\nVlJsL+CFctueBfau6Xkr07cv7LUXXH99Xb+yiIiIQA1Cg5ntYmZLgBXAzcDR7j6rkuLtgQXlti0A\nWptZ80zPXZ0//AFeeglmzKjrVxYREZGatDTMIvon9ATGAneb2Y51WqsaOuaYGIZ5443ZromIiEjj\n0zTTA9x9FTC79Ok7ZtYTOIPov1DefKBduW3tgMXuvqK6c40cOZI2bdqU2VZQUEBBQUGF5TfYAEaM\ngFGj4KqrYIstqjuDiIhIw1VYWEhhYWGZbcXFxfV2PvNaDjcwsxeBL9z9NxXsuwo4xN27p2y7H9jE\n3Q+t4jV7AEVFRUX06NEjo/p89x106AADB0b/hrZtMzpcRESkQZs+fTr5+fkA+e4+vS5fO9N5Gq4w\ns15mtm1p34Yrgf2Be0v3X2lm41MOuQXoZGZXm9kOZnYacCxwXV29gfI23RSuvhomTIDttoNLL62v\nM4mIiKxfMu3TsAUwnujX8AIxV0N/d59Uur890CFR2N0/Bw4DDgLeJYZanuLu5UdU1KkRI+Dzz2HY\nsLhVMWlStYeIiIhINWp9e6I+1Ob2RCp36Nkz+jq8/nrM5yAiItKY5cztiYbGLKaYnjoVnn4627UR\nERFp2Bp1aADo3x/22w8uvFBTTIuIiNRGow8NiSmm33kHnnoq27URERFpuBp9aADo3Rv+539g3Lhs\n10RERKThWi9CA8BJJ8Hjj8P332e7JiIiIg3TehMaBg+GVavgwQfj+bRpcdtCRERE0rPehIb27eHg\ng+Huu2HePDj88OgcOW1atmsmIiLSMKw3oQHiFsXUqREeWrSAbbeF0aOzXSsREZGGYb0KDUceCa1b\nw5w58OijMXPkgw/CwoXZrpmIiEjuW69CQ8uWcOed0SEyPx9+8xto0gTuuCPbNRMREcl961VoADj2\nWOjXL77fdFMYMgTGjo1OkiIiIlK59S40lDdiBHz5Jdx6a7ZrIiIiktvW+9DQvTsMHw5//CPMmpXt\n2oiIiOSu9T40AFx7LXTsGLcqfvop27URERHJTQoNQKtWcO+98N578Otfw3/+k+0aiYiI5B6FhlK7\n7x4TPz3/PHTtGt+LiIhIkkJDioIC+OijGF1xyimwaFG2ayQiIpI7FBrK2XxzGDMG3OGhh7JdGxER\nkdyh0FCBzTeHAQOin4OIiIgEhYZKDBkS61TMnp3tmoiIiOQGhYZKHHUUbLQR3HdftmsiIiKSGxQa\nKtGqFRx9dNyicM92bURERLJPoaEKJ5wAH38Mr7yS7ZqIiIhkn0JDFQ48EPbcE047DVasyHZtRERE\nsiuj0GBm55nZNDNbbGYLzOxRM+uaxnFDzOxdM1tmZl+b2Z1mtmnNq71uJJbN/vRTuPzybNdGREQk\nuzJtaegF/B3YEzgI2AB4zsxaVnaAme0LjAduB3YCjgV6ArfVpMLr2i67wHnnwZVXwvvvZ7s2IiIi\n2ZNRaHD3Q939Hnef6e7vAycDHYH8Kg7bC5jj7mPc/Qt3nwLcSgSHBuH882GHHeCXv4Rvvsl2bURE\nRLKjtn0aNgEc+K6KMlOBDmZ2CICZtQOOA56s5bnXmebN4ckno19D//7wXVXvVkREpJGqcWgwMwNu\nAF5z9w8rK1fasnAC8KCZ/QR8A3wPjKjpubNh221jMav586F375i/QZ0jRURkfWJew0kIzGwscDCw\nr7tX2mhvZjsBzwN/A54DtgSuBd5y92GVHNMDKOrduzdt2rQps6+goICCgoIa1bkuzJgBZ58NL7wA\nm20GvXpBfj6ceCJ06JC1aomIyHqosLCQwsLCMtuKi4uZPHkyQL67T6/L89UoNJjZaOAIoJe7z62m\n7N1AC3c/PmXbvsCrwJbuvqCCY3oARUVFRfTo0SPj+q0Ls2bF8tlvvglvvRXrVbzxRnwVERHJlunT\np5Ofnw/1EBoyvj1RGhiOAvpWFxhKtQJWldu2hugLYZmeP1fsuCNccQW8+CK89x4sWwZHHgnLl2e7\nZiIiIvUj03kabgaGAIOBZWbWrvTRIqXMFWY2PuWwicBAMzvVzLYvbWW4EXjT3efXwXvIuu22gyee\niPBw3HHw7bfZrpGIiEjdy7Sl4VSgNfAy8HXK4/iUMlsC/7277+7jgTOB04H3gQeBmcDAmlY6F+2+\nO0yYAK+/Dt26wYMPll2zYs0a+Oqr6BPxzjvZq6eIiEhNNc2ksLtXGzLcfWgF28YAYzI5V0N0yCEw\ncyacfjoMGgQXXAAnnwyLFsHDD8PXXyfLvvYa7Ltv1qoqIiKSMa09Ucfat4dHHolFrvbZJ/o9PPQQ\nDBwIjz8enSU7dozVM0VERBqSjFoaJH29e8fj9tuhadNYxyKhoCC233gjNGuWvTqKiIhkQi0N9ax5\n87KBAWDIkJhV8rnnslMnERGRmlBoyIJdd43HffdluyYiIiLpU2jIksGD4bHHYOnSbNdEREQkPQoN\nWVJQEBNBDR8e4eGHH7JdIxERkaopNGTJttvCqFHw8sux5PYOO8QcDiIiIrlKoSGLLr4Y5s2DTz+N\nxa769IEpU7JdKxERkYopNOSAn/881rDYdVfo1y9ChIiISK5RaMgRbdrA009D27ZwzjnZro2IiMja\nFBpyyIYbxgyS//wnvPpqtmsjIiJSlkJDjhk8GPLz4ayzYpErERGRXKHQkGPy8uDaa+Gtt+CBB7Jd\nGxERkSSFhhzUpw8ccwycfTYsXhzbFi+OEKHWBxERyRaFhhx1/fVQXBxzOSxfDkccERNCPfNMtmsm\nIiLrK4WGHNWxI/z5z/D3v8OAAXG7Yttt4Y47sl0zERFZXyk05LCRI6Fr15jwacIE+OMfYeJEmD9/\n7bK6bSEiIvVNoSGHNWsGTz4Jr7wChx4aIyuaNoXx42P/6tXwr3/BAQfEEtwDB8Lzz4N7dustIiKN\nk0JDjtt+e9hnn/j+Zz+D446LWxRFRTE08+ijYcWKuJXx0UfQvz8cfDB89VV26y0iIo2PQkMDM2xY\nTDO9xx7xfOpUeP11uOgieP99eOIJ+Pe/Y0rq++9Xq4OIiNQdhYYGplcvOOEEuOyy6By5117JfWZw\n2GERHg46CIYMiVaKN95I77Xdo7Vi0aL6qbuIiDRsCg0NjBnccw+cfz5ssEHFZdq2hYceikWwSkqg\nd2+YOze5/403YNKk5POVK6MFo1072HFHOPLI+n0PIiLSMCk0NGIHHBCdKPPy4OGHY5s7nHgiHH54\ncjXNG26Au+6K4HDJJXHL4913s1dvERHJTRmFBjM7z8ymmdliM1tgZo+aWdc0jmtmZpeb2edmVmJm\ns83s5BrXWtLWujUccgg8+GA8f+MN+OSTGIXx29/C7NkxgdQZZ8RiWeefD1ttBbfemt16i4hI7sm0\npaEX8HdgT+AgYAPgOTNrWc1xDwN9gaFAV6AA+CjDc0sN/epX0f9h9uwYrtmhAzzyCLz8cty62Gwz\nuPTSKJsIE/feC0uWZLXaIiKSYzIKDe5+qLvf4+4z3f194GSgI5Bf2TFmNoAIG4e6+0vuPtfd33T3\nqbWpuKTv8MOhZUu4++5Yv+LEE6FfPzjllBiaOXo0bLRRsvywYfDjj3Dffdmrs4iI5J7a9mnYBHDg\nuyrKHAG8DZxjZl+a2Udmdo2ZtajluSVNG20UweGqq2I9i5NOiu033QQvvbR2x8dttom1LsaO1UyT\nIiKSVOPQYGYG3AC85u4fVlG0E9HSsDPwS+AM4FhgTE3PLZk7/viYBGqffaBLl9jWqlWsqFmRM8+E\n996LhbNERESgdi0NNwM7AYPSOMcaYLC7v+3uzwBnAieZWfNanF8ycOihMaTytNPSK9+7d6x1ce65\n8Oab6Z/niSdi1IYmlRIRaXzMa/Db3cxGE7cdern73GrKjgP2cfeuKdt2BD4Aurr7ZxUc0wMo6t27\nN23atCmzr6CggIKCgozrLHGrIS+DmLhyZUwmtWABvPMObLJJbH/55eg4+fDDMSdEwurVsPPOMUHU\nxx8nWzRERKR+FBYWUlhYWGZbcXExkydPBsh39+l1eb6MQ0NpYDgK2N/dZ6dR/rfA9cAW7v5j6baj\ngAnARu6+ooJjegBFRUVF9OjRI6P6Sd36/HPYZRc466yYwwFitskXX4zWi4kTk0HkgQcgkefuuSdm\nrqwt95jQSkRE0jN9+nTy8/OhHkJDpvM03AwMAQYDy8ysXemjRUqZK8xsfMph9wPfAneZWTcz6w38\nFbizosAguWW77WDo0OgUWVISE0K9+GJ0pnzqKbjmmii3Zg385S8wYEC0MGRyS6MiX30Fu+0Gf/1r\nrd+CiIjUkaYZlj+VGC3xcrntQ4G7S7/fEuiQ2OHuy8ysHzG/w1tEgHgQuKgG9ZUsOOMMGDMmhmB+\n9FGstjl2LGy9NVxwAcycGXM9fPgh3HlnlK1NaJg3D/r2hc8+i5Ef55xTd+9FRERqLqPQ4O7Vtky4\n+9AKtn0MHJzJuSR3dO4cQzD/9jf4z3+ilaFly7hd0aQJPPZYjLQYMCAW0Hr77ZiBsqQEWlQzsLao\nKPpFbLddPF+6NEZ0rF4Nv/893H579K2obJ0NERFZd7T2hKRl5MhoUVi0CH73u9jWtGl0iJwxAxYu\nhAkTYnvPnvFBP2NG2ddwLzvL5KJFcOCB0V8i4eWXY+bKJ56IYaLLl8MHH9TrWxMRkTQpNEha9t8f\nevSI2wbduq29f/PNYcMN4/vu3aFZs7VvUYweHetavP9+PL/kkphs6qWXkpNIvfxyTC61887wi19E\nS8a0afX2tkREJAMKDZIWM3j22WRrQlWaN48P/PKh4f774/bDL38ZK2mOHRu3Pb7/Ptkq8corEVDM\nYvKpXXeNdTNERCT7FBokbZttBptuml7ZPfcsGxrmz4/no0bBDz9EMOjQIRbGatEiWhsWL4bp02Nf\nwh57qKVBRCRXKDRIvdhzzxj9sGhRPJ84MVoPRoyITpJNmsC118bS3fvtB5MmwWuvxW2K1Kmte/aE\nf/8bli1LbnOHu+6KCadqauVKreIpIpIphQapF3vtFV8ffzy+PvZYhIPNNovJob7/HgYOjH0HHACT\nJ8f8D1tuGaM1Enr2jCAxPWV6kosvht/8Bk4+OdkX4uuvYdCg9DpNzpkD+fnRb+KHH2r7TkVE1h8K\nDVIvOnWK2SHPOiumlH7hBTjqqOT+1KGYffvGX/133pnsz5Cw007Rt2HatGhhuOyyGLFxwgkxzDMR\nSs4+O1ow+vaNlonKTJoEu+8eLRfFxfCHP9Tt+xYRacwUGqTejB4d4aBv31hhMzU0pNp9d9h44/gQ\nL7/qZtOm0SowYUIsonXRRREa7rknyl56aYy4KCyMeSS23jrOV1GLw/TpcNhhMQrkrbfghhtg/Phk\n8BARkaopNEi92XRTuOOOuHWw887w859XXK5p0wgEULYTZMKee8Ibb0TrwNNPR3AA+POfo1/DwIGw\n997RavBi3nVXAAAc+klEQVTii3EL5Pe/L/saixbB0UfHOhoTJ0bdTj4ZDj88Jqvae++Ytvr002M+\nilTffx9zU7z2Wq1+HEDMOzFvXu1fR0QkGxQapF4ddhhcfjlceGHV5QYOjGCxww5r7zvnHHj++Zhp\ncsCA5PY+fWIVzu+/j1aNvLwIA5ddFrchpk6NcqtWRX+H5cvhn/9M3hoxi1Bz/PEx90TPnvDII3FL\npH//mGDqgw9i++23R8goKanZz2H6dDjmmAg0XbrESBERkYamRktj1zetcinpmj07JotKvfWxZk20\nKHTqFK0Kp58Ot90WwaNv36pfb8WKWPL7xhsjpECEiL/+NeaXuOQSOP/8zOu5337R4jJgQMxP8fbb\ncdtFRKSu5cwqlyK5plOntftK5OXFQlpPPhm3FcaOhVtuqT4wQExMdcIJ0fFy6lS47rr4ethhcfvj\nsstg7tzM6rhsWbzeWWdFqwvEaqGZWLUqApKISDYpNEij9KtfRR+KO+6A886DYcMyO94sho2OHBlz\nSUD0odhkExg+PD7E0zV1aswL0adPrBDati188klm9bnzzrh98+OPmR0nIlKXFBqkUWraFG69NT7o\nL7usbl5z443jw/u55+CUU5JzRJQ3e3bsX7Einr/8cqzNsdNO8bxz58xbGl55JfpTaPEuEckmhQZp\ntA48MPog5NXhv/JDDonhnvfcEx0ojz02FthKXZNjzBj4xz+ibwREaOjTJzn/RJcumYeGRKfOd9+t\n7TuoX0uXwpdfZrsWIlJfFBpEMjRoUIymeOklWLAgbl/89a+xb/XqmDMC4O9/T/ZnSJ1/ItOWhvnz\n4fPP4/vyy43nmksuWXuuDRFpPBQaRGrglFPg22/h1Vfhiitisqh33olWhW++iY6Y06bFKIxEf4aE\nzp0jbKS79kWilaFXr9wPDW+9FWuOzJ+f7ZqISH1QaBCppcMPh622ij4U990XHTAvuQS22y7Wydh8\n85gHIiGxtkb51oYpU+CMM9buZDl1atwCOeywCA2V9aXINvfk7RMtZy7SOCk0iNRS06YxOuO++2Jy\nqCFDYhXP005LtjKkrqfRpUt8TQ0NU6bAwQfDTTetPa311KkxkqN792idSNyqSHjkkVhCvC4njCop\nga++yuyYuXNjKnBQaBBprBQaROrAsGExHHLx4ggNECtxtm4dnSdTbbppDL1MhIZp02LSpx49Ihzc\ncEOy7E8/xURQe+8doQHK3qJYtQrOPTfKXHrp2vX68cfkYl+ZuOiiqM/q1ekfk6hX9+5xztpatChC\nl4jkDoUGkTrQoUNMMrX33tC1a2xr2xa++CLWtigv0RnSPfpHdOsWk1GdfXb0k0gsBT5jRvzVv/fe\n0L49bLFF2REUEybE6/zqV9F/YtassucZPjzW7ujbN9k3IuG779beBhFE7rkHFi4suyR5dcFjxowI\nREcfHS0NtZlsdvbsmLhrt91i2XQRyQ0KDSJ15J574Jlnym7bZJOKh3x27hwTPL36aizlffnlsNFG\nETy23TYCAMSHerNm8Ve/WfwVn/iLfs2a6IR58MEwbhx07Bh9IhIf1lOmwN13x22S776DffaJ+R4S\nLr44wsTSpWXr9vzz0VGzadP4HuI199yz4taMhBkzon577hnnq+kMlqtXxzofm24aLTX771/1eUVk\n3VFoEKkjG26YnD2yOom5GsaMiUW6DjwwtjdtCiNGwAMPxEqd118fgaF589i/227J0PDkk7HuxgUX\nxCJc118fE0+NHBm3JUaMiPUtbropWid22CEmp4L4YH744ZiAKhEMEu6+O2afPOSQ5L7p06P1YNy4\nylsQEqFh993jeU1vUdx4Y4Sp8ePh9dejJWbMmNq1XIhI3VBoEMmCzp1jaOY//xktAakdJYcNi1sR\nzz4bH8B/+UtyX/fu0RHyttuiXK9e8QA44ogIDmPHwvbbxxDQ0aOjU2ZeHgweDI8+GoHi1VdjWOTG\nG5fteFlcDP/6F5x4IvTrFx/ay5ZFK0qTJjBnTrxueUuWRAjq3j1W8uzUqWadIWfPjgXB/vCHaGHI\ny4Mjj4xbJems+bFqFfzf/0V4EpG6l1FoMLPzzGyamS02swVm9qiZdc3g+H3NbKWZ1emqWyINTWLY\nZfPma/d52GST6Avx0UfRGnDQQcl9ic6Q//u/0Ls3PPhgcp9ZfNi+9VYM0Tz99OhYmTB4cNyKmDgR\nHnoobmecdlq0WCQ6PE6YEK0PQ4ZEaFi5MpYZLyyM/hFt2yZnv1y8OALNd99Fi0dq/Xr2jJaGZcvg\nmmvg5pvTmynywQejtSWxsBfEyBBIr+Vi7NgISocdFq0itbFkicKHyFrcPe0H8BTwa6AbsCvwBPA5\n0DKNY9sAnwJPA9OrKdsD8KKiIhdpjP7zH3dw/9//zey4VavczzrL/amnanbenj3dDz3UffPN3c8+\n2/3116Mer78er7377u79+kXZNWvct9nGfdddo0xRkfspp7h37hz7Tj01th96qPvo0e5Nm7qXlMSx\nf/ube/Pm7h06uDdrFvvA/YAD3D/4IMrMnet+5pnub72VrN8++7j/8pdr17tDh6hvVebPd2/TJur4\n29/G+W68sWY/J3f3UaPczeJaiTQkRUVFDjjQwzP4jE/nUbuDYTNgDbBfGmULgUuAUQoNsr5bs8b9\nhhvcv/xy3Z73xhvjfz24T5sWQWGzzdzPPdf94ovd8/LcJ09Olh86NMp26xZ1fvrpeD56dHwdPDg+\nWDfbLMJFwltvxf7DDnP/7DP37793v/de9x12cN9gA/djjolQAcmQsnBhvNadd65d72OPdd9//+Tz\nZcuiPqlOPtl9003dFy2Kfaed5r7hhu7ffpvez+bNN91XrEg+32WXqN/Eiekdn21ffOHeu3cylMn6\nK5dDQ2dgNbBTNeWGAm8Qt0MUGkSy5JtvIhh06pT80D355Gh5yMuLv65T3X9//Ja44op4vmKF+yab\nxLa9947Qcckl8fyEE8oe+9VXa3+wl5TEOTp1iq+J8PHxx+7jx8f38+evXe+//jUCwKpV7sXFEVJu\nvjm5/+2349hbb01uW7jQvWXLCEPV+eqrCCx/+Us8/+ijZLg699zqj88FV18d9f2f/3FfvjzbtZFs\nysnQAFjp7YlXqinXBfgG+Hnpc4UGkSz6v/8r+4H7yCPxm6BXL/eVK8uW/eEH94KCsh/kJ50UrQX/\n/nc8X73a/Q9/cH/uuczrsnx5tA6ceWa0JvTsWXG5l1+OOr7/frK1JD8/uf+3v3XfdtsIFal+/3v3\nn/3MfcmSquuRCEdbbBF1uvJK91at3A85JP56rw/p3PaYMsX97rvdH3ssWmyqsu++7rvtFi04Z5xR\nN3WUhqk+Q4O512wck5mNBQ4G9nX3byopk1fawnCHu99Wuu1i4Eh371HFa/cAinr37k2bNm3K7Cso\nKKCgoKBGdRaRtZWURIfG006Drbeuvvw338SiVPvtVzfnP/vsWEp81Sr405/gwgvXLrN0KbRpE+t7\nXH01LF8e01zPnBnzWrRvH3NUlJ/PYe7cWAvk6qvhzDNjbouK5s049dQYWfKf/8TIlFtvjdfdZ58Y\n0rp4MWywQc3f44QJMSQ3MTvolVfGKJFXX63857hkSXRoTUwP3rFjdJCtyH/+Ez+D22+Pn9UZZ0Qn\nzn790qtfSUkM25WGp7CwkMLE0rqliouLmRyzouW7e90OPKhJ0gBGA18AHasp14bo8/ATsLL0sTpl\nW59KjlNLg8h64pNPkrcC3n238nK77BKdMMF90qTo9HjhhclWgk8/rfi4oUOj1aBdu7gF889/rl1m\nxx2jU+rRR0fnT4jXnTo1vn/zzSg3Zoz7ddetfdsl1cKF7hMmRAuMe7QSmCVv4Zx1VnzftKn7RRcl\nj/vsM/eHHko+Hz066vvxx+7/+EccU1nrxLhxcY7586Nuv/iF+/HHV17HVLfcEj+f995Lr3xNXXtt\ntOBUZebMaHWq6ucr1cup2xOlgWEe0CmNsgbsVO4xBviQGIFR4agLhQaR9Uv//u4dO1b9YfGb38Rv\nrD32iHKnnBJ9I/r1c99vv8qPmzvXfcQI90svdd95Z/fDDy+7f/78ZEhIjCZp1iz6TqxYEc3911/v\nPnt23JZJfPiXlETfh7vucl+6NPl6J5/s/+0E+tJL7httFGFk3Lhkf5DrrnM/6ij3Pn2Sx51ySux7\n+eV4fzvuGB1G3ePDNBGWKnLMMe577ZV8PmpU3JYpf7upvFdfjfCSl+d+3HFVl62t/Hz3tm2rrtMf\n/xjv8+2367cujV3OhAbgZuB7oBfQLuXRIqXMFcD4Kl5DfRpEpIwvv4z+ClW55Zb4jXXvvfF80qRk\nC8Xtt6d3nptuig/+1BEVDz0Ur/HVV/G8V6/kh7V7BJLjjosWi3bt4q/+5s3dW7dOnv+886LsggUR\nOI4/PjprJjomJvpUfP11cnTKtde6t2gRwWTNGvettooP786d3R9/PBkg3OODtkWLGHFT3vLl0Uk0\n0VnVPdlC8vrrlf8s5s2L97P//tHHxSzZT6WurVmT/Hmljs4pb489oswFF9RPPdYXuRQa1pTeXij/\nODGlzF3ApCpeQ6FBRDK2cGGMZEgMi1y9Om4ltGgRHTbTkRg9cscdyW2nnebepUvy+dKl7j/+mHz+\npz/FrZC8vOS8D2++GX8VP/543G5o2TJCx6WXxvfffhtBaOTIGApZkTffjN/AU6bEbRmI2x/Nm8ft\ngl13Ldvykp8frS3lJYbBpoauVauig2nq7Y/yBg5033rrCDorVkRLz6BBlZevjURrDsTPqyLFxfEz\nbt3afaed6qce64ucCQ3r6qHQICLpuPNO96uuyuyYAw5wP+ig5POdd3YfNqzy8v/6V/ym3Gabiocy\n/vBDfEAPHerevn36E3b99FOEg6uvjlaCDTeMWx5XXhnnu+22suWHDo3Jt1KtWuXet29ywq1Ugwat\nXT7h00+jZeGWW5Lbxo6NbddcE99X1b8kU5Mnx3vaZ5+K6+oeE5ZBnB/cZ82qu/OvbxQaRETqyG23\nxV+08+cnZ+a8557Kyy9cGH/9p7ZOlHfttcm/pDOZXOmAA9yPOCJuiRx1VGxbudL90UcjVKS6/vpo\nVUkdVjpqVLyXivo6jBsX9VmwYO19I0bE7ZPUFpWSEvc994zbN2bRF+Pjj9N/L6l++KHsz/SOO+I1\nJ0yIOn344drH/OlPcYtm2bIIU+U7Tb7zjvv552feSfLWWyM8ZbNz5cqV6/b8Cg0iInVk0aLo/Hfc\ncfFhDdFZsirVzamwfHlMdd2/f2Z1GTUqmuObNCk7MVVFXnyx7F/gzz8fH8SXXlpx+W++qTgQfftt\nfCiXn8gr1eLF0SKQn192lkz36Px5/vnRAbVnzyhb3uWXe5kRLeec477ddhFSWrWquHWoZ8+YE8Q9\n+pSkztlRUhKzkoL7jBmV17u8pUtj4rLENOh15aef4nZKOt5+O+b/GDu27s5fnfoMDVrlUkTWK23b\nxoqgjz0WC4bdcw906FD1MZttVvX+Fi1iRdD77susLr16xTwMq1cn53CozK67xtf33otFxYYOjcXM\nzj+/4vLt28MvfhFzRKxaldx+yy1xvtNOq/xcG28cy7O/9x6cd15y+/LlMGBAzGOxwQaxiNi//rX2\n8Q8/HF+nTo2vH38MXbtCy5bQv3/87FMtWQJFRdCnTzw/5ph47Tlz4vnll8cqqhtvXHaRtlRewZRD\nY8bADz/ARhuVXc21ts45J67H8uVVl3vjjVj2fuHCin9O6frkk1h6PifUdQqpiwdqaRCRerRiRTSD\nZ9vSpdHqscsu6ZXfcssYWXDLLdHKUN19/5tuir+yf/7zmCL78MNjdMfvfpfe+a67Lo4fNy6en39+\nHP/RR/G8V6+1W1cS826YRSdT9+g3cvrp8f0998T+J55IHpPozJl43e+/j9snm2wSI1OaNo2WkVNO\nifdSvqn/yivdu3Yt+9f/4sUxxHP48Ojf0aNHeu+5OiUl0YcFYnrzysyZE7d49tsvrtmGG659yyld\nvXvHjKeJuT+qo9sTIiKN1PHHR5+IdBx8sPuAAdHU/6tfpXfM22/HSImmTeND/tprq59WO2HNmugk\n2qRJrFy6wQZl1/K49dboU/HNN8ltV10VI0iOOy4+qFevjj4hiZEnq1dHP442bSIkrFkT0323b182\nDCxYECup5uXFaIqSkrglA2VXRn3xxeTkWanrhFx2WQScefOSE4DNmxfnGDq07MJoq1a5Fxam93NJ\nTLver1/MhfH99xWXSwzvLS5OjpSZOrX61y8vsS5KVfN0lKfQICIi/538KNN7++7p/5Va3sqV0UkT\nYmhq6giSb7+ND8bU+SN23z1Cyi23RNj44IM49umnk2V++CFWPO3SJYIFROfMinz6aTKUrFyZXNbd\nPba3axejYS64IMLJnDnuzzwT9TrzzCj33XdRl7Fjo1MmlF2VNTEvxr77Vt9X4YgjYj6Jr7+O/hmJ\nOTrKO+aYCGmJem+0UfUzYlbkxhvjvXTsGOu+uEcL1RFHxPusiEKDiIj8t2k/MdJiXfnxx7i9kJhO\nO9VRRyWHds6ZE/V74IEINYmJmiqa5nvWrGhdOOSQCBTphprhw6PT6f33x6yZ7dvHSJglS+L2zT77\nREvH4YeXvR3Qt2982LduHbcyUuv061/HkNo2bWIESWWtBwsWRIvN6NHx/Pzz41xff1223OrVa8+T\nccgh0VJUmdmzy7agJOy7b7yXSy+NWxxLliTD4xZbVNxJV6FBRET8s8/igy2Xpll++OH4JPnb32J6\n7RYtoj/BqlXuG28cfyFvsEH1U1qnK7HiKcSHcOp8EnfdFdv337/scFL3ZP+MLbeM2xQtWsScECUl\nESRGjYqf689+5j5kSMXnvv76uOWxaFE8T8zRceqpZcslAlPq7YSrr668X8OaNRFomjcvewtj7lz/\n7wiYRCAbOTJaTc44I85d0YRcCg0iIuLuubeY0/LlyUW+yneyPOig2L7jjnV3vtWrY9rrd96peN+D\nD1Z8i2HOnPjQfvzxeH7kkdEqkbg1kZhCOzHJVeJ5SUkEooEDo2PmwIFlX/eaa+JDPHVOixtvjJ9F\nanCZNs3/269hzZqys5i+8krs23bbuN2SmEX0uusiSCTeT9++UW6XXaIz7333xfMLL4zWj+uui1Ci\n0CAiIjlr9eqKw8xFF8WnzBFHrPs6VSS1teOuuyIc9OtXdtrqFSuio+nAgfGeTjopWkr69IkpsL/8\nsuxr/vhjhKbUjqlHHx0jHsqfe+ON3X/722ghyctznzgx9h1xRASBBQsiOHTpEuGrU6eyt6IKCyOg\nTJkSz9esiQXSzCKkbLhhBAzN0yAiIjkrLw/M1t6+997xtWvXdVufyjRtmvz+iCOi3s8/D8cdl9ze\nrBn8+c/wyCMwbBiMHw933QUvvQTXXgtbb132NVu2hIsvjvkjpk2DNWvglVeSc06knrtXL7j99ph3\noVcvGDwYHn0UJk6Es86CLbaAJ5+Ezp1j3opmzeD005OvMWgQzJ+f/LmaRd1Wr465O5Yuhdat6/In\ntjaFBhERqRd77QVNmsBOO2W7Jmtr2xZ6947vU0MDwK9/DV26wD/+ARdcAEOGVP1aJ50UE2n17x/B\n4rvv1g4NEOHi5pvhww8jKHTsGBNZbbklFBREmZ13hqeegrffhpkzoV+/sq9R0URjFQW2+tK0+iIi\nIiKZ+9nP4sOvW7ds16RiI0bA5pvHB3Wqpk3hzjvhhRdg1KjqX6dp02iJOOGEmC2yefNka0CqPfaI\nB0SZxx6DffeFc8+N5w2BQoOIiNSb3XbLdg0qd8wx8ahIr17xSFebNhECrroKfvwxphavzs9/Dl9+\nWfa2Sa5rQFUVERHJXXl5la8FUpmGFBhAfRpEREQkTQoNIiIikhaFBhEREUmLQoOIiIikRaFBRERE\n0qLQICIiImlRaBAREZG0KDTIOlFYWJjtKkgd0vVsXHQ9JV0ZhQYzO8/MppnZYjNbYGaPmlmVS5GY\n2dFm9pyZLTSzYjObYmb9a1dtaWj0S6lx0fVsXHQ9JV2ZtjT0Av4O7AkcBGwAPGdmLas4pjfwHHAI\nseT1S8BEM+ueeXVFREQkWzKawNLdD019bmYnAwuBfOC1So4ZWW7TBWZ2FHAEMCOT84uIiEj21LZP\nwyaAA9+le4CZGbBxJseIiIhI9tV4qYzSD/8bgNfc/cMMDv0jsCHwUBVlWgDMnDmzptWTHFNcXMz0\n6dOzXQ2pI7qejYuuZ+OS8tmZxlqbmTF3r9mBZmOBg4F93f2bNI8ZDNwKHOnuL1VT7r4aVUxEREQA\nhrj7/XX5gjUKDWY2muiT0Mvd56Z5zCDgDuBYd3+mmrJtiUDyOVCScQVFRETWXy2A7YBn3f3bunzh\njENDaWA4Ctjf3WeneUwBERh+5e5PZFxLERERybqM+jSY2c1AAXAksMzM2pXuKnb3ktIyVwBbu/tJ\npc8HA+OA3wNvpRyz3N0X1/4tiIiIyLqQUUuDma0hRkuUN9Td7y4tcxewrbsfUPr8JWKuhvLGu/tv\nMq+yiIiIZEONO0KKiIjI+kVrT4iIiEhaFBpEREQkLTkXGszsdDObY2bLzewNM9sj23WS6pnZKDNb\nU+7xYbkyl5rZ12b2o5k9b2ads1VfKcvMepnZ42b2Vem1O7KCMlVePzNrbmZjzGyRmS0xswlmtsW6\nexeSqrpramZ3VfB/9qlyZXRNc0C6i0Wui/+jORUazOxXwN+AUcAviLUpnjWzzbJaMUnXv4F2QPvS\nx36JHWZ2DjAC+B3QE1hGXNtmWainrG1D4F3gNCro7Jzm9bsBOAwYSHR+3gp4pH6rLVWo8pqWepqy\n/2cLyu3XNc0N1S4Wuc7+j7p7zjyAN4AbU54b8CXwp2zXTY9qr90oYHoV+78GRqY8bw0sB47Pdt31\nWOtarSFmbU37+pU+XwEcnVJmh9LX6pnt97S+Pyq5pncB/6ziGF3THH0Am5Veh/1Stq2T/6M509Jg\nZhsQq2W+mNjm8a5eAPbOVr0kI11Km0I/M7N7zawDgJltT/wVk3ptFwNvomub89K8frsT876klvkI\nmIuucS7rU9rcPcvMbjazTVP25aNrmqvKLBa5Lv+P5kxoIJJTE2BBue0LiB+G5LY3gJOJ6b9PBbYH\nJpvZhsT1c3RtG6p0rl874Cdfe8I2XePc9TRwInAA8Cdgf+Cp0sUIIa6brmmOqWSxyHX2f7TGq1yK\npHL3Z1Oe/tvMpgFfAMcDs7JTKxGpjLunrjT8gZm9D3wG9AEqXVBQsu5mYCdg32ycPJdaGhYBq4k0\nlKodMH/dV0dqw92LgY+BzsT1M3RtG6p0rt98oJmZta6ijOQwd59D/B5O9LjXNc0xpWs/HQr08bKr\nS6+z/6M5ExrcfSVQBByY2FbaDHMgMCVb9ZKaMbONiF8+X5f+MppP2WvbmugJrGub49K8fkXAqnJl\ndgA6AlPXWWWlxsxsG6AtkPgw0jXNISmLRfb1cqtLr8v/o7l2e+I6YJyZFQHTgJFAK2LBK8lhZnYN\nMJG4JbE1cAmwEnigtMgNwIVm9imx5PlfiJExj63zyspaSvuedCb+WgHoZGbdge/cfR7VXD93X2xm\ndwLXmdn3wBLgJuB1d5+2Tt+MAFVf09LHKGK43fzSclcTrYPPgq5pLrE0FotkXf0fzfbQkQqGkpxW\n+oaXE+ln92zXSY+0rlth6T/Q5URv3PuB7cuVuZgYFvQj8Yupc7brrcd/r83+xNCr1eUe/0j3+gHN\nibHki0p/IT0MbJHt97a+Pqq6pkAL4BkiMJQAs4GxwOa6prn3qOQ6rgZOLFeu3v+PasEqERERSUvO\n9GkQERGR3KbQICIiImlRaBAREZG0KDSIiIhIWhQaREREJC0KDSIiIpIWhQYRERFJi0KDiIiIpEWh\nQURERNKi0CAiIiJpUWgQERGRtPw/LFUKEVoHO5UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1acb7c24400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 网络示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了示例，我们给网络输入一个字母并询问下一个字母是什么，下一个字母再作为下下个字母的预测输入，直到输出EOS token\n",
    "- 创建输入分类的Tensor, 初始字母和空的隐藏状态\n",
    "- 输出 `output_name` ，包含初始的字母\n",
    "- 最大输出长度,\n",
    "    - 将当前字母输入网络\n",
    "    - 获取最大可能输出，和下一个的隐藏状态\n",
    "    - 如果字母是EOS，则停止\n",
    "    - 如果是一般字母，则加到`output_name`，继续\n",
    "- 返回最后的姓名单词\n",
    "\n",
    "> 另一种策略是不需要给网络决定一个初始字母，而是在训练时包含**字符串开始**标记，并让网络选择自己的初始字母"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ramanovov\n",
      "Uarin\n",
      "Shavani\n",
      "Garen\n",
      "Eren\n",
      "Roure\n",
      "Sangara\n",
      "Pare\n",
      "Allan\n",
      "Ollang\n"
     ]
    }
   ],
   "source": [
    "max_length = 20\n",
    "\n",
    "# sample from a category and starting letter\n",
    "def sample(category, start_letter='A'):\n",
    "    with torch.no_grad(): # no need to track history in sampling\n",
    "        category_tensor = categoryTensor(category)\n",
    "        input = inputTensor(start_letter)\n",
    "        hidden = rnn.initHidden()\n",
    "        \n",
    "        output_name = start_letter\n",
    "        \n",
    "        for i in range(max_length):\n",
    "            output, hidden = rnn(category_tensor, input[0],hidden)\n",
    "            topv, topi = output.topk(1)\n",
    "            topi = topi[0][0]\n",
    "            if topi == n_letters -1:\n",
    "                break\n",
    "            else:\n",
    "                letter = all_letters[topi]\n",
    "                output_name+=letter\n",
    "            input = inputTensor(letter)\n",
    "            \n",
    "        return output_name\n",
    "    \n",
    "# get multiple samples from one category and multiple starting letters\n",
    "def samples(category, start_letters='ABC'):\n",
    "    for start_letter in start_letters:\n",
    "        print(sample(category, start_letter))\n",
    "        \n",
    "samples('Russian', 'RUS')\n",
    "\n",
    "samples('German', 'GER')\n",
    "\n",
    "samples('Spanish', 'SPA')\n",
    "\n",
    "samples('Irish', 'O')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
